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Abstract 

Integrated  Computational  Materials  Engineering  (ICME)  model  verification  and  validation  (V&V)  is  difficult  because 
materials  processing,  microstructural  evolution,  and  property  development  contain  a  rich  mix  of  length  and  time 
scales  with  an  equally  complex  set  of  interacting  phenomena  and  mechanisms.  Beyond  these  difficulties,  engineers 
who  adapt  these  models  rarely  generate  independent  validation  data  sets  to  confirm  model  adeguacy,  guantify 
uncertainty,  and  identify  potential  error  sources.  Even  when  a  validation  data  set  is  produced  and  applied,  the  range 
of  model  applicability  is  limited  by  the  range  on  input  model  parameters  contained  within  the  data  set.  In  this 
paper  we  provide  a  summary  of  a  recommended  approach  to  ICME  V&V  and  include  descriptions  of  V&V  planning 
checklists,  an  ICME  Tool  Maturity  Level  assessment  guide  and  examples  of  how  such  practitioner  aids  might  be 
employed. 


Background 

The  implementation  of  Integrated  Computational  Mate¬ 
rials  Engineering  (ICME)  offers  potential  for  significant 
benefits  in  all  aspects  of  aerospace  materials  and  pro¬ 
cesses  engineering  —  including  materials  design  and  de¬ 
velopment,  process  modeling,  and  prediction  of  material 
behavior.  There  is  potential  to  significantly  reduce  cost, 
time  and  risk  for  new  materials  and  process  insertion,  to 
optimize  material  processing  and  properties  for  specific 
applications,  and  to  computationally  integrate  materials 
and  processing  with  other  engineering  disciplines  [1]. 
Considerable  effort  has  been  expended  on  model  devel¬ 
opment  over  the  past  three  decades,  and  while  many  of 
these  computational  models  are  now  in  use,  significant 
challenges  to  comprehensive  development  and  imple¬ 
mentation  of  ICME  remain  [2]. 

One  of  the  key  areas  identified  for  future  effort  is  the 
verification  and  validation  (V&V)  of  ICME  methods  and 
models  [2,3]  —  including  the  need  to  develop  guidelines 
and  standards  for  this  critical  activity.  The  Air  Force  Re¬ 
search  Laboratory  sponsored  an  assessment  of  V&V  ac¬ 
tivities  in  2010  and  2011,  with  the  objective  of 
developing  a  recommended  approach  and  practice  for 
ICME  V&V.  This  assessment  included  a  review  of  rele¬ 
vant  V&V  activities  in  other  engineering  disciplines,  and 
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the  development  of  a  proposed  general  approach  and 
guiding  philosophy  suitable  for  ICME.  During  this  as¬ 
sessment,  it  became  clear  that  some  specific,  pragmatic 
guidance  on  how  to  assess  and  plan  ICME  V&V  activities 
would  be  useful,  possibly  even  essential,  to  aid  materials 
and  process  engineering  practitioners.  Several  “practi¬ 
tioner  aids”  were  developed  including  planning  and  exe¬ 
cution  checklists  with  instructions  and  examples,  a 
recommended  approach  for  Tool  Maturity  Level  assess¬ 
ment,  and  an  approach  to  assess  “risk  vs.  consequences” 
of  ICME  application  [4,5].  These  references  are 
appended  as  “Additional  files  1  and  2”.  This  assessment 
and  the  practitioner  aids  were  intended  specifically  for 
aerospace  ICME  applications,  and  were  designed  to  be 
compatible  with  other  gated  review  processes  for  tech¬ 
nology  and  product  development  commonly  used  in  the 
aerospace  industry. 

The  need  for  ICME  V&V 

Engineering  disciplines,  such  as  Fluid  and  Solid 
Mechanics,  have  developed  and  validated  mathematical 
and  computational  models  and  frameworks  of  greater 
maturity  compared  to  those  available  today  for  ICME 
practitioners.  A  systematic,  rigorous,  and  disciplined  ap¬ 
proach  to  verify  and  validate  computational  models  and 
methods  was  deemed  essential  by  these  other  disciples,  to 
confirm  the  accuracy  of  model  predictions.  These  disci¬ 
plines  have  consequently  pursued  community-wide 
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efforts  to  establish  model  verification  and  validation 
guidelines  [6,7].  Notably,  representatives  of  these  disci¬ 
plines  have  asserted  that  the  development  and  application 
of  their  guidelines  were  of  paramount  importance  for 
building  trust  among  both  technical  customers  (e.g., 
materials  application  engineers  and  product  design  engi¬ 
neers)  and  managerial  decision  makers  (e.g.,  product 
managers  and  ICME  project  sponsors). 

The  motivation  and  rationale  for  ICME  verification 
and  validation  differ  little  from  that  of  these  other  disci¬ 
plines.  However,  owing  to  its  relative  immaturity,  ICME 
has  only  recently  begun  to  gain  acceptance  as  a  powerful 
method  for  enhancing  materials  analysis,  development, 
and  implementation.  The  application  of  verification  and 
validation  will  accelerate  the  maturation  of  ICME  by 
sharpening  focus  on  the  critical  needs  of  its  customers 
and  associated  decision-makers.  V&V  will  help  ICME 
practitioners  to  more  clearly  define  of  these  needs  and 
pursue  improved  model  development  trajectories,  illumi¬ 
nated  up-front  by  ICME  accuracy  and  uncertainty 
requirements.  By  determining  these  metrics,  the  ICME 
community  and  customers  alike  will  have  quantitative 
information  describing  the  maturity  level  of  ICME  tech¬ 
nologies,  promoting  further  acceptance  and  application 
benefit. 

The  importance  and  future  value  of  ICME  to  the 
materials  science  and  engineering  community  is  broadly 
recognized,  and  is  now  the  focus  of  the  “Materials 
Genome  Initiative”  recently  announced  by  the  White 
House  Office  of  Science  and  Technology  Policy  [8]. 
Hence,  the  development  and  implementation  of  a 
standard  approach  for  ICME  V&V  is  especially  timely, 
even  urgent,  as  increased  focus,  investment,  and  expec¬ 
tations  are  directed  toward  ICME. 

Methods 

The  approach  taken  to  develop  this  recommended  prac¬ 
tice  followed  three  guiding  tenets: 

1.  Utilize  current,  recognized  practices  for  V&V  as  the 
basis 

2.  Facilitate  alignment  with  established,  gated  review 
practices  for  product  and  technology  development. 

3.  Provide  simple,  useful  guidance  and  tools  to  aid 
practitioners  in  planning  and  assessing  V&V 
activities  and  results. 

The  basis  for  ICME  V&V 

We  have  found  that  the  ASME  Guide  10-2006,  “Guide 
for  Verification  and  Validation  in  Computational  Solid 
Mechanics”  [7]  provides  a  sound  basis  for  guiding  ICME 
V&V  activities,  and  have  subsequently  used  and  refer¬ 
enced  it  extensively.  It  should  be  noted  that  the  develop¬ 
ment  of  this  document  required  extensive  effort  by  a 
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number  of  participants  over  a  period  of  many  years.  The 
Guide  provides  an  excellent  set  of  definitions  and  guide¬ 
lines,  and  is  applicable  to  a  broad  range  of  computational 
disciplines  and  applications. 

It  is  useful  to  ensure  common  understanding  of  the 
terms  “verification”  and  “validation”  for  this  application 
[7]: 

•  Verification:  the  process  of  determining  that  a 
computational  model  accurately  represents  the 
underlying  mathematical  model  and  its  solution. 

•  Validation:  the  process  of  determining  the  degree  to 
which  a  model  is  an  accurate  representation  of  the 
real  world  from  the  perspective  of  the  intended  uses 
of  the  model. 

Verification,  in  this  context,  means  ensuring  that  com¬ 
puter  code  performs  as  intended  and  equations  are 
solved  correctly.  Validation,  in  this  context,  means  asses¬ 
sing  the  extent  to  which  a  model  represents  the  “reality 
of  interest,”  over  the  range  of  intended  application.  Key 
guidance  from  the  ASME  V&V  Guide  that  we  regard  as 
especially  important  V&V  of  ICME  methods  and  models 
is: 

•  Verification  must  precede  validation;  and  when  used, 
calibration  must  precede  validation. 

•  The  need  for  validation  and  the  specific 
computational  accuracy  requirements  depend  on 
intended  use,  and  should  be  considered  as  part  of 
the  V&V  activities. 

•  Validation  of  any  complex  system  should  be  pursued 
hierarchically. 

•  Simulation  results  and  experimental  data  must  be 
generated  independently  and  have  assessment  of 
uncertainty  to  be  meaningful. 

There  is  extensive  discussion  and  illustration  of  the 
recommended  processes  for  verification  and  validation, 
including  these  specific  items,  in  ASME  V&V  Guide 
2006,  and  by  Thacker  [9].  In  addition,  these  references 
stress  the  importance  of  Uncertainty  Quantification 
(UQ)  as  an  essential  focus  area  for  V&V.  While  UQ  is 
expected  to  be  especially  critical  for  ICME  V&V  and  the 
subject  of  future  publications,  it  is  not  addressed  in  any 
detail  in  this  document. 

Alignment  of  ICME  V&V  with  product  and  technology 
development  processes 

Successful,  broad  implementation  of  ICME  in  the  future 
will  span  an  enormous  range  of  applications.  ICME 
encompasses  all  areas  of  materials  and  process  engineer¬ 
ing,  from  early  development  of  technology  through  prod¬ 
uct  support  in  the  field.  The  application  of  ICME  may 
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involve  simple  trending  or  guidance  for  development,  or 
activities  as  complex  and  critical  as  life  prediction  in 
safety-critical  systems.  The  consequences  of  using  ICME 
for  decision-making  will  vary  accordingly.  Further,  the 
guiding  philosophy  proposed  for  ICME  V&V  should  also 
align  with  established  product  development  processes, 
such  as  Integrated  Product  Development  (IPD)  and  the 
US  DoDs  Defense  Acquisition  Milestones  [10],  and  tech¬ 
nology  development  processes  such  as  Technology  and 
Manufacturing  Readiness  Levels  —  TRL  and  MRL,  re¬ 
spectively  [10,11].  These  are  all  “gated  review  processes” 

—  meaning  that  there  are  rigorous  criteria  for  assessing 
completion  of  a  particular  gate,  and  that  progressive 
stages  imply  more  substantial  commitment  and  invest¬ 
ment.  These  processes,  or  similar  ones,  have  been 
adopted  by  various  companies  and  agencies,  so  they  are 
broadly  recognized  and  widely  used.  The  level  of  matur¬ 
ity  of  a  particular  technology  —  for  a  specific  application 

—  is  now  readily  communicated  by  simply  stating  its 
current  “TRL  status.” 

Similarly,  ICME  methods  and  models  would  benefit 
from  a  Tool  Maturity  Level  (TML)  assessment  process, 
which  could  be  used  to  guide  development  and  applica¬ 
tion  of  ICME  methods  and  models  analogous  to  the  TRL 
ranking  process.  If  accepted  and  broadly  used,  as  is  the 
TRL  process,  a  Tool  Maturity  Level  would  readily  convey 
the  state  of  maturity  of  a  particular  ICME  model  or 
method,  and  offer  objective  guidance  on  where  in  the 
TRL  or  IPD  process  its  use  would  be  appropriate.  Finally, 
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TML  criteria  could,  and  should,  be  considered  in  light  of 
the  potential  decisions  or  consequences  of  application. 
These  assessments  are  critical  steps  in  determining  the 
level  of  ICME  V&V  needed  for  a  specific  application, 
and  are  instrumental  in  planning  and  executing  V&V 
activity. 

Guidelines  and  practitioner  aids  for  ICME  V&V 

The  philosophy  and  basis  for  the  recommended  ap¬ 
proach  to  ICME  V&V  are  relatively  easy  to  describe  and 
reference  —  but  somewhat  more  difficult  to  implement 
in  a  consistent  and  sustained  manner.  Consequently, 
some  simple  tools  for  V&V  planning  and  assessment  that 
enable  practitioners  to  get  started  on  ICME  V&V  imme¬ 
diately  are  essential.  The  most  important  tools  for  this 
are  ICME  V&V  System-Level  and  Model  Checklists,  a 
Tool  Maturity  Level  (TML)  assessment  guide,  and  an  ap¬ 
proach  for  assessment  of  specific  application  risks.  Initial 
versions  of  these  tools  have  been  drafted  in  spreadsheet 
format,  along  with  a  simple  set  of  process  steps  to  guide 
practitioners  in  their  use  [4,5].  The  recommended 
process  for  using  these  tools,  in  its  simplest  form,  is  pre¬ 
sented  in  Figure  1. 

This  is,  of  course,  a  simplistic  representation  of  an  ef¬ 
fective  V&V  planning  and  execution  process,  but  illus¬ 
trates  the  logical  sequence  of  activities  needed,  and  that 
in  fact  iteration  may  be  required  to  arrive  at  an  accept¬ 
able  ICME  V&V  plan  for  a  specific  application,  including 
any  risk  mitigation  activities  that  may  be  indicated.  A 


Figure  1  Flow  chart  of  simplified  ICME  V&V  planning  and  assessment  process. 

V  J 


Cowles  et  al.  Integrating  Materials  and  Manufacturing  Innovation  2012,  1:2 
http://www.immijournal.eom/content/1  /I  /2 


more  detailed  process  description  is  presented  in  the 
referenced  reports  [4,5]. 

These  tools  will  facilitate  a  consistent  approach  to 
ICME  V&V,  and  help  ICME  developers  and  practitioners 
bridge  the  critical  step  from  general  guidelines  and  pro¬ 
cedures  to  their  specific  applications.  We  expect  that 
these  tools  will  be  refined  and  improved  significantly 
with  time  and  experience. 

V&V  for  special  causes 

There  is  significant  potential  for  ICME  to  improve  our 
understanding  of  new  materials  and  processes,  and  re¬ 
duce  risk  for  early  applications,  if  we  employ  a  rigorous 
V&V  approach  and  utilize  the  TML  and  risk  assessment 
approaches  outlined  here  to  consider  and  assess  the  pos¬ 
sibility  of  infrequent  or  rare  events.  These  are  sometimes 
described  as  “special  causes,”  especially  when  referring  to 
product  failure  events. 

ICME  practitioners  and  customers  gain  confidence 
when  V&V  efforts  succeed  in  demonstrating  that  a  mod¬ 
eling  system  accurately  describes  the  “physical  reality”  of 
interest,  quantifies  modeling  uncertainty,  and  clearly 
defines  the  range  of  applicability.  Sometimes,  ICME  goals 
are  modest,  for  example  when  the  system  is  intended  to 
screen  alloy  compositions  or  process  trajectories  in  sup¬ 
port  of  detailed  research  planning.  For  such  low-risk 
cases,  modeling  and  subsequent  V&V  can  safely  adopt  a 
narrow  focus  on  the  most  dominant  and  expected  pro¬ 
cessing  and  behavioral  mechanisms.  Correspondingly,  de¬ 
fining  customer  needs,  the  reality-of-interest,  accuracy/ 
uncertainty  requirements,  and  consequent  tool  maturity 
level  is  straightforward.  Conversely,  the  difficulty  of  de¬ 
fining  and  confirming  acceptable  ICME  system  capabil¬ 
ity  increases  significantly  when  goals  are  both  stringent 
and  complex  and  the  required  tool  maturity  level  is  ne¬ 
cessarily  high  in  order  to  address  the  later  stages  of  the 
TRL  process.  Examples  of  this  latter  category  include 
ICME  systems  that  support  production  qualification  of  a 
new  process  or  the  determination  of  minimum  proper¬ 
ties  of  a  new  material  for  use  by  life  management 
engineers. 

A  late  TRL  material  system  that  involves  “special 
causes,”  a  subset  of  the  latter  high-risk  category,  poses  a 
significant  challenge  to  ICME  and  V&V  —  much  as  it 
has  frustrated  todays  empirical,  data-driven  materials  en¬ 
gineering  approaches.  Special  causes  produce  anomalous 
process  defects,  microstructures,  and/or  material  behav¬ 
ior  that  are  difficult  to  detect  via  experiment  or  test  be¬ 
cause  their  frequency  of  occurrence  is  typically  low  and/ 
or  their  full  impact  is  unknowable  a’priori  based  solely 
on  experience.  As  a  historical  example,  consider  the 
introduction  of  hot  isostatically  pressed  (HIP)  powder 
metallurgy  (PM)  superalloys  in  the  1970s.  At  the  time  it 
was  well  known  that  these  As-HIP  PM  alloys  were  defect 
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sensitive  and  that  there  were  a  multiplicity  of  different 
types  of  low-cycle-fatigue  (LCF)  initiation  sites  among 
failed  test  bars.  However,  one  defect  type,  produced  by 
organic  powder  contaminants  whose  gasification/oxida¬ 
tion  weakened  interparticle  bonds,  ultimately  led  to 
abandonment  of  As-HIP  PM  processing.  Despite  an  un¬ 
precedented  number  of  mechanical  tests,  the  Achilles 
heel  of  As-HIP  superalloy  processing  wasn’t  fully 
revealed  until  an  engine  failure  gave  the  ultimate  verdict. 

The  difficulty  posed  by  “special  causes”  upon  ICME  and 
V&V  is  significant,  but  these  anomalies  also  offer  a  great 
opportunity  for  advancing  the  benefit  of  ICME  via  V&V. 
Unlike  traditional  data-driven  methods,  ICME  and  V&V 
can  adopt  a  proactive  approach  to  analyzing  and  modeling 
“special  causes”  including  both  anomalous  processing  arti¬ 
facts  and  material  behavioral  issues.  The  role  of  V&V  is  to: 

•  Focus  up-front  on  both  the  explicit  and  implied 
requirements  of  the  customer. 

•  Exhaustively  examine  the  conceptual  model  to 
assure  that  all  relevant  physics  is  included  —  both 
mechanisms  for  nominal  behavior  and  exceptional 
mechanisms  associated  with  “special  causes”. 

•  Exercise  the  ICME  modeling  system  to  evaluate 
process  and  material  behavior  outside  the  nominal 
processing  window  to  assess  process  robustness  and 
evaluate  the  influence  of  processing  discrepancies. 

•  Proactively  analyze  the  likelihood  and  effect  of 
“special  causes”  through  detailed  modeling  or 
approximate  methods. 

•  Exercise  risk  management  to  identify,  assess,  and 
abate  material  and  processing  “special  causes” 
outside  the  purview  of  the  ICME  team  owing  to 
technical  or  programmatic  constraints. 

These  concepts  are  mentioned  subsequently  in  the  en¬ 
suing  sections  of  this  paper  that  discuss  the  details  of  the 
proposed  V&V  approach.  We  now  offer  a  simple  hypo¬ 
thetical  example  to  make  some  aspects  of  the  outlined 
approach  more  concrete. 

A  gas  turbine  engine  manufacturer  has  undertaken  de¬ 
velopment  of  a  new  product  with  a  very  aggressive 
schedule.  The  stringent  schedule  is  deemed  acceptable 
because,  although  the  engine  is  significantly  larger  than 
its  predecessor,  it  will  use  no  new  materials.  However, 
the  turbine  design  manager  questions  whether  estab¬ 
lished  design  minimum  properties  for  the  cast  and 
wrought  disk  alloy  can  be  achieved  in  the  larger  stage  1 
turbine  disk.  The  manager  conveys  his  concerns  to  the 
ICME  team  and  requests  an  analysis  of  disk  heat  treat¬ 
ment  and  a  projection  of  mechanical  properties  because 
the  larger  forgings  won’t  be  available  for  property  testing 
for  another  year.  The  team  drafts  a  conceptual  model  fo¬ 
cused  on  heat  treatment,  precipitation,  and  mechanical 
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property  modeling.  However,  during  V&V  review  of  the 
conceptual  modeling  plan,  one  ICME  engineer  notes  that 
the  larger  disk-forging  billet  will  require  a  larger  cast 
ingot  outside  of  prior  experience.  The  engineer  expresses 
concern  that  solidification  of  an  ingot  this  size  would  in¬ 
fluence  dendrite-arm-spacing,  grain  structure,  the  scale 
of  interdendritic  segregation,  and  even  could  lead  to 
freckle  segregation,  a  “special  cause”  condition.  Also  dur¬ 
ing  V&V  risk  assessment,  the  heat  treatment  modeler 
questioned  the  validity  of  assumed  quench  heat  transfer 
coefficients  because  the  larger  disk  requires  a  larger 
quench  tank  with  a  different  agitation  system.  The  ICME 
team  then  conducted  analyses  to  assess  the  likelihood  of 
ingot  freckling  and  estimate  quench-process  robustness. 
Based  on  the  teams  findings  they  extended  their  analysis, 
worked  with  the  suppliers  to  confirm  and  resolve  these 
issues,  and  thereby  avoided  a  significant  schedule  disrup¬ 
tion  during  early  production  stages  of  the  program. 

In  conclusion,  although  focused  up-front  consideration 
and  ICME  predictive  analysis  of  “special  causes”  will  in¬ 
crease  the  likelihood  of  early  detection  and  resolution  of 
some,  a  few  will  undoubtedly  escape  early  identification. 
However,  even  for  these  latter  cases,  ICME  and  V&V  can 
be  applied  to  more  quickly  assess  and  resolve  such  sur¬ 
prises  when  they  do  occur. 

ICME  V&V  checklists 

Although  the  Verification  and  Validation  Checklists 
below  were  developed  as  practitioner  aids  for  aerospace 
materials  researchers  we  believe  that  they  are  applicable 
within  the  broader  materials  community.  The  checklists 
are  intended  to  provide  guidance  to  ICME  development 
teams,  particularly  those  with  limited  experience  with 
V&V.  In  formulating  the  checklists,  an  expansive  view  of 
V&V  has  been  adopted  that  extends  beyond  the  tech¬ 
nical  strategies  and  techniques  necessary  to  perform 
V&V.  If  a  major  goal  of  V&V  is  to  build  trust  in  ICME  sys¬ 
tem  predictions,  it  is  paramount  for  a  V&V  team  to  under¬ 
stand  the  intended  use  of  the  system  as  well  as  the  needs 
and  expectations  of  those  who  rely  on  ICME  predictions 
—  the  technical  customers  and  managerial  decision¬ 
makers.  For  this  reason,  non-technical  activities  have  been 
included  in  the  checklists,  as  described  below. 

Introduction  and  precepts 

The  V&V  precepts,  underlying  the  V&V  checklists,  chal¬ 
lenge  some  common  shortcomings  of  materials  modeling 
and  include  the  following: 

Understanding  customer  needs 

•  An  ICME  model  should  serve  a  real  customer  need 
by  supporting  material  and  process  design, 
application,  and  support  decisions. 
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•  ICME  predictions  must  have  sufficient  accuracy  to 
meet  this  need.  For  low  risk  decisions,  even  a  model 
that  provides  proper  trends  may  be  useful. 

•  The  customer /decision-maker  must  trust  the 
modeling  results.  Decision-makers  trust  sources  of 
information  and  analysis  methods  that  are  widely 
accepted  and  have  proven  reliable.  V&V  should 
substantiate  that  ICME  predictions  provide  equal  or 
better  confidence  than  historical  precedents. 

•  Dialog  with  the  customer  needs  to  continue 
throughout  the  ICME  V&V  process  to  address 
identified  issues  and  risks  that  bear  on  the  implied  as 
well  as  explicit  customer  needs. 

ICME  V&V  is  a  continuous  process 

•  ICME  V&V  should  begin  at  program  start  and 
continue  until  program  completion.  By  doing  so, 

V&V  teams  define  customer  needs,  application 
scenarios,  and  accuracy  requirements  early.  Armed 
with  these  inputs,  the  ICME  team  can  better  define 
the  ICME  model,  identify  modeling  risks,  identify 
validation  plans,  and  design  validation  uncertainty 
quantification  analyses.  This  precept  counters  the 
too  common  notion  that  V&V  should  be  considered 
only  after  ICME  model  development  has  been 
completed.  Indeed,  too  many  traditional  materials 
modeling  programs  iteratively  carryout  model 
refinement  and  never  attempt  formal  validation. 

V&V  activities  promote  team-wide  communications 

•  Productive  ICME  V&V  demands  integrated  product 
development  (IPD)  teamwork  and  communications 
among  participants  having  diverse  backgrounds  and 
expertise.  ICME  V&V  practitioners  inform  and 
engage  other  stakeholders  (material  specialists, 
application  engineers,  product  engineers,  and  other 
customers)  about  modeling  goals,  approaches, 
limitations,  and  risks,  including  those  associated  with 
“special  causes”. 

V&V  risk  assessment  and  uncertainty  quantification 

improves  model  performance 

•  V&V  encourages  early  identification  of  unintended 
side-effects  from  inappropriate  modeling 
assumptions,  formulation  of  more  realistic  boundary 
conditions,  assessment  of  alternative  modeling 
approaches,  recognition  of  shortcomings  with  model 
input  data,  and  resolution  of  implementation  issues. 

•  Focused  attention  of  ICME  risks  fosters  the  early 
identification  of  material  processing  and  behavioral 
mechanisms  that  allow  higher  fidelity  predictions 
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that  more  closely  simulate  the  reality-of-interest. 
Such  inclusiveness  enables  prediction  of  off-nominal 
materials  and  processes  including  some  “special 
cause”  events. 

•  Model  improvement  requires  that  the  ICME 
development  team  quantify  and  understand  model 
uncertainty  and  errors;  this  includes:  i.)  Model 
parameter  sensitivity  analysis,  ii.)  Quantification  of 
model  input  errors  and  variation,  iii.)  Analysis  of 
uncertainty  propagation  throughout  the  modeling 
system,  and  iv.)  Uncertainty  analysis  to  quantify 
model  fidelity  during  model  validation. 

Description  of  the  ICME  V&V  checklists 

V&V  ICME  system  and  model  level  checklists  were  con¬ 
structed  to  provide  a  “quick  start”  listing  of  traditional 
V&V  activities  that  describes  “what”  should  be  consid¬ 
ered  and  addressed  during  V&V  activities  without  man¬ 
dating  “how”  these  activities  should  be  carried  out.  As 
such,  the  V&V  checklists  are  not  prescriptive  in  that  each 
ICME  team  can  determine  which  elements  of  a  checklist 
are  relevant  and  helpful  to  their  project.  The  checklists 
were  designed  to  be  straightforward  and  simple  to  fill 
out  and  maintain.  These  V&V  checklists  will  evolve  as 
ICME  teams  gain  greater  V&V  experience. 

The  ASME  V&V  guide  [7]  advocates  that  V&V  should 
be  approached  hierarchically  within  a  modeling  system 
(i.e.,  from  components,  through  subassemblies,  to  full 
systems).  This  recommendation  promotes  a  systematic 
approach  that  helps  identify  individual  model  inadequa¬ 
cies  and  cumulatively  builds  confidence  in  the  overall 
modeling  system.  The  proposed  ICME  V&V  guidelines 
agree  with  this  concept  but  also  acknowledge  that  mate¬ 
rials  systems  usually  involve  interacting  model  networks 
that  are  more  complex  than  the  tree-like  morphology 
common  to  the  mechanical  disciplines.  A  simplified 
(and  incomplete)  ICME  example  is  shown  in  Figure  2. 
It  remains  important  to  decompose  such  complex 
ICME  modeling  systems  into  meaningful  constituent 
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subsystems.  Given  the  benefits  of  hierarchical  analysis, 
at  least  for  large  ICME  systems,  two  checklist  templates 
were  developed  —  one  for  individual  models  and  a  sec¬ 
ond  for  the  ICME  system  and  application.  The  ICME 
checklists  are  summarized  in  Table  1  that  lists  both  the 
categories  and  associated  checklist  items;  this  table  sepa¬ 
rates  system  from  model  elements.  Notice  that  the  system 
checklist  deals  primarily  with  ICME  customer  needs, 
business  case,  and  systemic  considerations  and  risks, 
whereas,  the  modeling  V&V  checklist  focuses  upon 
model  development,  technical  risks,  verification,  and  val¬ 
idation.  The  model  checklist  attempts  to  adhere  to  the 
process  flow  advocated  by  the  ASME  V&V  guideline  [7]. 
The  complete  checklists  are  available  in  Microsoft  Excel™ 
format. 

Tool  maturity  level  (TML)  and  application  risk  assessment 

The  TML  Assessment  Guide  was  intended  to  facilitate 
assessment  of  maturity  and  capability  relative  to 
intended  ICME  applications,  and  to  provide  a  useful  tool 
for  guiding  and  assessing  ICME  V&V  activities  during 
model  development  or  refinement  when  integrated  with 
the  ICME  V&V  Checklists. 

Currently,  there  is  no  simple,  standard,  gated  review 
process  in  broad  use  for  assessment  and  communication 
of  the  maturity  level  of  an  analytical  model  or  tool  —  at 
least  not  in  the  same  sense  as  the  TRL  process.  Sandia 
National  Laboratories  and  NASA  have  published 
approaches  and  standards  for  predictive  capability  and 
maturity  of  computational  models  and  simulations 
[12,13].  Both  are  excellent  references  and  guidelines.  The 
Sandia  report  describes  levels  from  zero  to  three  in  ma¬ 
turity,  and  contains  very  specific  criteria  descriptions  for 
various  assessment  elements.  NASA-STD-7009  describes 
levels  from  zero  to  four  (5  levels),  and  has  more  general 
descriptions  which  would  be  adaptable  for  various  sys¬ 
tem  level  applications.  Interestingly,  the  NASA  standard 
includes  practitioner  (personnel)  capability  and  training 
as  one  of  the  major  assessment  elements. 
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Figure  2  Interaction  between  process,  microstructure  and  properties  within  an  ICME  system. 
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Table  1  Outline  of  the  V&V  Checklists 


ICME  System  and  Application  Checklist 

SI:  Definition  of  Customer  Needs 

•  Identified  customer,  ICME  opportunity,  and  benefits 

•  Established  &  refined  business  case  for  the  ICME  system 

•  Defined  ICME  system  accuracy  and  uncertainty  requirements 

S2:  ICME  Modeling  System  Formulation 

•  Formulated  the  ICME  system  architecture  and  constituent  modules/models 

•  Selected  software  platform(s)  and  integration  strategy  and  tools 

•  Identified  and  resolved  computation  and  implement  issues 

S3:  ICME  Modeling  System  Risks 

•  Formulated  overall  project  risk  management  plan 

•  Identified  and  abated  system  and  model  development  risks 

S4:  ICME  Modeling  System  Uncertainty  Quantification 

•  Identified  and  engaged  uncertainty  quantification  expert 

•  Established  project-wide  UQ  methods,  tools,  and  application  approach 

.  Formulated  system  level  error/uncertainty  propagation  analysis  strategy 

S5:  ICME  Modeling  System  Level  V&V 

•  Established  hierarchical  system  wide  validation  strategy 

•  Tested  and  verified  the  integrated  modeling  system 

•  System  level  validation  modeling  and  experimental  design  completed 

ICME  Model  Verification  and  Validation  Checklist 

Ml:  ICME  Model  Development 

•  Established  detailed  modeling  approach 

-  Developed  mathematical  model  and  initial  computational  model 

-  Conducted  sensitivity  studies  to  assess  inputs,  internal  parameters  &  BC/ICs 

-  Performed  UQ  to  determine  output  uncertainty  based  on  inputs  uncertainty 

-  Participate  in  system  level  uncertainty  propagation  analysis  activities 

•  Established  detailed  experimental  approach  to  support  model  development 

-  Established  plan  to  measure  internal  parameters,  inputs,  and  outputs 

-  Conducted  experiments  to  measure  internal  parameters,  inputs,  and  outputs 

-  Applied  UQ  to  assess  accuracy  and  variation  for  experimental  results 

M2:  Experimental  Support  of  Model  Development  and  UQ 

•  Established  experimental  approach  to  support  model  development  &  UQ 

•  Determined  experimental  methods  and  sources  of  uncertainty 

•  Experiments  conducted  and  UQ  applied  to  determine  uncertainty  of  results 

•  Assess  data  and  uncertainty  in  support  of  system  level  validation 

M3:  Model  Verification 

•  Established  model  verification  plan 

•  Identified  verification  benchmark  model  and/or  data 

•  Checked  and  executed  computational  model  to  identify/fix  coding  problems 

•  Compared  model  results  against  benchmark(s) 

•  Identified  and  repaired  computation  model  deficiencies 

M4:  Model  Validation 

•  Established  overall  validation  plan 

•  Defined  and  executed  experimental  plan  for  validation 

-  Analyzed  results  using  UQ  methods 

•  Defined  and  executed  modeling  plan  for  validation 

-  Analyzed  results  using  UQ  methods 

•  Applied  UQ  methods  to  determine  model  accuracy  &  range  of  applicability 

•  Completed  activities  and  support  to  system  level  validation 

What  we  propose  for  ICME  use  is  based  on  a  simple 
five-level  maturity  assessment  as  presented  by  Morris 
[14],  at  the  8th  International  HCF  Conference  in  2003. 
This  approach,  adapted  for  ICME,  is  illustrated  in 
Table  2. 

The  concept  is  simple,  useful  for  ICME  V&V  process 
development,  and  provides  an  approach  to  align  ICME 
V&V  with  technology  and  product  development  require¬ 
ments.  The  assessment  elements  are  briefly  described  in 
Table  3.  The  assessment  elements  and  gate  criteria,  while 
developed  specifically  for  ICME  applications,  are  gener¬ 
ally  consistent  with  those  in  both  the  Sandia  report  and 
the  NASA  standard  previously  referenced. 

Determining  what  tool  maturity  level  is  required  for  an 
ICME  model  or  method  —  and  consequently  the  level 


and  fidelity  of  verification  and  validation  of  a  particular 
analytical  tool  —  is  of  course  dependent  upon  the  spe¬ 
cific  application.  The  maturity  requirements  will  very 
likely  increase  as  the  region  of  ICME  application  pro¬ 
gressed  in  the  technology  or  product  development 
process.  This  is  not  to  say  that  “high  TML”  ICME  tools 
are  not  needed  or  desired  in  early  stages  of  technology 
or  product  development  -  but  rather  that  the  later  stages 
require  higher  TML  levels  because  the  potential  conse¬ 
quences  of  application  generally  become  more  severe. 

Since  there  will  likely  be  many  specific  ICME  applica¬ 
tions  in  the  future,  developing  a  very  prescriptive  guide¬ 
line  regarding  TML  requirements  is  neither  likely  nor 
desirable.  Consequently,  it  is  useful  to  consider  a  supple¬ 
mental  means  of  assessing  “risk  vs.  consequences”  of 
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Table  2  Tool  Maturity  Level  (TML)  Description  for  ICME 

(Adapted  from  Morris,  [14]) 

Tool  Level  -  ICME  Tool  Maturity  Description 

TML 

1  Analytical  process  is  exploratory  in  nature.  Fidelity  of 
predictions  is  largely  unproven.  Provides  some  physical 
insight,  but  cannot  reduce  development  testing. 

2  Proven  capability  for  comparative  assessment,  ranking  or 
trending.  Experimental  validation  is  still  necessary.  Can 
drive  development  or  assessment  plan  and  test  matrix. 

3  Material  or  process  can  be  developed  or  assessed  with 
significantly  reduced  testing.  Expectation  that 
development  iterations  will  be  reduced  or  eliminated. 
Accuracy  and  uncertainty  effects  must  be  quantified. 
Range  of  applicability  well  defined. 

4  Material  or  process  performance  and  impact  on  system  or 
application  are  understood.  Accuracy  and  uncertainty 
effects  must  be  verified.  Additional  data  may  be  required 
when  applied  to  new  materials  or  processes,  or  to  extend 
range  of  application. 

5  All  material  and  process  performance  and  system 
interaction  effects  are  understood  within  defined  range  of 
application.  Analytical  process  can  be  applied  without 
testing. 


ICME  application,  and  using  the  outcome  to  determine 
whether  the  V&V  status  of  a  specific  ICME  model  is  suf¬ 
ficient.  An  excellent  example  of  a  “Risk  vs.  Consequence” 
table  was  developed  by  NASA  [15]  and  is  reproduced  in 
Figure  3.  The  associated  text  boxes  and  arrows  indicate 
how  it  relates  to  ICME  model  and  V&V  considerations, 
and  where  in  the  TRL  or  IPD  process  a  specific  ICME 
tool  might  be  applied. 

This  tool  is  used  in  conjunction  with  the  Checklists 
and  TML  assessment  for  ICME  development  and  valid¬ 
ation  planning.  A  “risk  vs.  consequences”  assessment  is 
intended  to  provoke  an  assessment  of  deficiencies  or 
risks  that  should  be  addressed  in  conjunction  with  spe¬ 
cific  ICME  applications.  This  allows  ICME  tool  develo¬ 
pers  to  objectively  assess  their  potential  impact  and 
develop  risk  mitigation  actions  as  warranted. 

Our  long-term  vision  is  that  the  TML  Assessment 
becomes  a  “gated  process,”  where  expectations  are  that 


all  prior  TML  criteria  are  met  before  a  particular  analyt¬ 
ical  model  or  tool  can  progress  to  the  next  level  but 
where  flexibility  is  maintained  to  permit  tailoring  the  as¬ 
sessment  to  the  many,  diverse  applications  which  ICME 
will  surely  encounter  in  the  future. 

Results  and  Discussion 

Verification  and  validation  examples  are  needed  to  help 
practitioners  accelerate  and  strengthen  the  application  of 
V&V  within  ICME  development  programs.  Of  course,  the 
best  and  most  relevant  examples  will  ultimately  be  drawn 
from  successful  ICME  projects  involving  an  integrated  sys¬ 
tem  of  models.  However  for  now,  a  simple  ICME  example 
is  offered  that  illustrates  some  of  the  V&V  concepts  and 
checklist  elements.  The  example  is  based  upon  research  by 
one  of  the  authors  involving  implementation,  verification, 
and  validation  of  a  single,  publically  available  precipitation 
model  for  6000  series  aluminum  alloys. 

Customer  Needs  (Checklist  SI)  —  An  ICME  develop¬ 
ment  team  identified  the  need  to  establish  a  model  to  de¬ 
scribe  precipitation  during  the  aging  of  aluminum  A6082 
thin-wall  extrusions;  it  would  be  used  provisionally  within 
a  larger  integrated  multi-model  ICME  system.  The  cus¬ 
tomer  for  the  precipitation-modeling  project  required 
rapid  identification  and  implementation  of  a  model  cap¬ 
able  of  simulating  A6082  hardness  trends  as  a  function  of 
heat  treatment  process  variables.  The  model  was 
expected  to  benefit  the  overall  ICME  development  effort 
by  providing  guidance  for  development  of  a  final,  more 
comprehensive  precipitation  model  and  allowing  earlier 
ICME  system  studies.  Although  model  accuracy  require¬ 
ments  were  modest,  the  customer  and  development  team 
agreed  that  verification  and  validation  were  required  to 
assure  that  the  model  produced  reasonable  trends  and  to 
provide  data  that  could  be  used  in  subsequent  verification 
of  the  more  comprehensive  final  model. 

The  precipitation  model  and  associated  V&V 

The  ICME  development  team  identified  a  model,  devel¬ 
oped  by  Myhr  [16,17],  for  implementation  as  the 


Table  3  Tool  Maturity  Level  (TML)  Assessment  Elements 


Assessment  Category 


Description  and  Criteria  Considerations 


Model  Basis  and  Definition 

Complexity  and  Documentation 

Supporting  Data 
Model  Verification 


Definition  of  the  model  and  its  basis,  intended  application,  input  variables  and  outputs,  ranges  of  inputs  and 
outputs,  sub-models  defined.  ICME  Systems  Level  Checklist  items. 

Model  or  method  flow  diagram,  assessment  of  sequential  or  interdependent  computations,  User's  Guide. 
ICME  V&V  Model  Checklist  items. 

Identification,  adequacy,  archiving,  and  documentation  of  supporting  data. 

Computer  code  and  model  verification,  version  control  and  documentation,  consistency  with  ASME  V&V 
Guide  10-2006. 


Range  of  Applicability  and  Range  of  applicability  and  range  of  all  input  parameters,  Uncertainty  Quantification  plan  for  model 

Uncertainty  Quantification  parameter  sensitivity  and  model  output  uncertainty,  limitations  defined. 


Validation 


Validation  plans  and  execution  for  sub-models  and  system  level,  benchmark  cases,  Risk-vs.-Consequence 
assessment,  risk  mitigation  requirements. 
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•  Risk*  associated  with  ICME  use 
can  be  related  to: 

♦  Model  complexity 

♦  Supporting  data 
♦ Verification  and  Validation 

Ideally  captured  by  :  ICME  Tool 
Maturity  Level. 

(* Risk  of  significant  error  or  failure 
of  ICME  methods  or  models) 

•  Consequences  associated  with 
ICME  use  can  be  related  to: 

♦  Specific  application 

♦  Location  in  TRL  or  IPD 
process,  if  applicable 

♦  Impact  on  product  or  system 


Consequences  vs.  Likelihood  5X5  Risk  Matrix 


Ref:  “Guidelines  for  Preparing  Project  Risk  Management  Plans,*  New  Millennium 
Program,  NASA,  NMP-GL-07,  June  7,  2005. 


Figure  3  Alternative  risk  analysis  approach  to  assess  ICME  tool  maturity  [15]. 
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provisional  precipitation  model  for  the  A6082  alloy.  This 
model  is  physically  based  and  applies  classical  precipitate 
nucleation  and  growth  theory  in  combination  with  Fee¬ 
ders  precipitate  strengthening  theory.  The  model  focuses 
on  the  dominant  |3”  strengthening  precipitate,  uses  prag¬ 
matic  thermodynamic  simplification,  and  relies  on  experi¬ 
mentation  to  calibrate  the  nucleation  rate  equation.  The 
Myhr  papers  describe  this  precipitation  model  in  full  detail 
along  with  associated  validation  experiments  and  results. 
Although  Myhr  et.  al.  verified  and  validated  the  precipita¬ 
tion  model,  the  case  study  model  required  additional  veri¬ 
fication  of  its  MatLab™  implementation  and  A6082 
validation  experiments  because  this  alloys  composition 
and  specific  processing  route  could  fall  outside  the  range 
addressed  by  Myhr  s  research. 

This  case  study  was  carried  out  prior  to  development 
of  the  V&V  checklists;  however  retrospectively,  the  im¬ 
plementation  of  the  Myhr  precipitation  model  for  A6082 
was  performed  in  accordance  with  key  elements  of  the 
AIAA  [6]  and  ASME  [7]  guidelines  as  well  as  the  V&V 
checklists.  The  ensuing  case  study  description  is  orga¬ 
nized  using  these  guidelines  and  reference  the  V&V 
checklist  where  appropriate. 

The  Sargent  diagram  [6,18],  Figure  4,  provides  an  over¬ 
view  of  the  V&V  approach  executed  within  the  case  study. 
The  study  followed  the  three  successive  steps  shown  in  the 
diagram,  including:  i.)  Confirmation  of  the  Myhr  model 
prior  to  implementation,  ii.)  Verification  that  the  mathem¬ 
atical  model  was  properly  implemented  within  MatLab™, 


and  iii.)  Validation  assessing  model  fidelity  and  uncertainty 
when  applied  to  aging  of  A6082. 

Model  Confirmation 

Assessment  of  the  precipitation  model  was  made  easier  be¬ 
cause  Myhr  had  published  a  comprehensive  description  of 
the  model  physics  and  mathematical  implementation  within 
a  peer  reviewed  journal  [16,17].  The  precipitation  model 
includes  a  heterogeneous  nucleation  law,  a  relationship  the 
Gibbs  free  energy,  a  linear  precipitate  growth  equation,  the 
Gibbs-Thompson  equation  for  precipitate  solubility,  and  a 
continuity  equation  governing  the  evolution  of  the  precipi¬ 
tate  particle  size  distribution.  The  model  also  includes 
strengthening  relationships  based  on  Friedels  formulation 
to  derive  alloy  yield  strength  and  hardness  values  based  on 
the  size  distribution  of  the  strengthening  precipitates.  While 
the  confirmation  exercise  ( Checklist  Ml)  concluded  that 
these  are  all  necessary  elements,  it  also  identified  several 
potential  weaknesses  in  the  model  formulation: 

•  Thermodynamics:  The  model  uses  simplified, 
approximate  thermodynamic  relationships  rather 
than  thermodynamic  software  such  as  ThermoCalc™. 

•  Nucleation  Rate:  The  model  does  not  account  for 
precipitate  incubation  or  the  variation  in  potency 
among  heterogeneous  nucleation  sites. 

•  Growth  Rate:  The  model  ignores  the  non-linearity  of 
compositional  profile  in  the  matrix  and  impingement 
effects. 
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Figure  4  Sargent  diagram  showing  overview  of  case  study  V&V  activities  [6,18]. 


These  issues  could  influence  accuracy  and  miss 
trends  associated  with  differing  cooling  rates  following 
solutionizing,  particularly  for  rates  that  are  much 
slower  than  those  studied  by  the  original  model  devel¬ 
opers.  Although  the  assessment  team  concluded  that 
rectifying  these  issues  would  exceed  the  scope  of  the 
study  and  lose  precious  time,  it  decided  to  track  these 
issues  within  the  project  risk  management  plan 
(Checklist  S3). 

Verification  of  the  Mat  Lab™  Computational  Model 

Verification  (Checklist  M3)  for  the  A6082  precipitation 
model  included  three  steps  that  involved  checking  the 
MatLab™  code  to  identify  and  resolve  errors;  comparing 
simulation  results  against  a  benchmark  computation; 
and  examining  model  output  for  anomalies.  The  original 
Myhr  papers  described  the  computational  implementa¬ 
tion  thoroughly,  thereby  reducing  debugging  to  little 
more  than  correcting  several  simple  code  syntax  and 
typographical  errors. 

The  benchmark  verification  entailed  comparing  out¬ 
put  for  the  MatLab™  implementation  against  results 
reported  in  Myhr  [16].  The  comparison  included  check¬ 
ing  predictions  of  nucleation  rate,  particle  number 
density,  mean  particle  radius,  and  the  particle  size  dis¬ 
tribution  at  discrete  times  during  isothermal  aging.  For 
each  output  characteristic,  output  from  the  MatLab™ 
implementation  matched  the  published  Myhr  results 
quite  well.  A  comparison  of  predicted  precipitate 


particle  size  distributions  is  shown  in  Figure  5.  The  two 
sets  of  modeling  results  nearly  overlay  each  other  ex¬ 
cept  near  the  peak  of  the  distribution;  this  discrepancy 
was  attributed  to  likely  small  differences  in  the  time 
and/or  radius  (particle  size)  increments  used  for  nu¬ 
merical  integration. 

The  benchmark  verification  demonstrated  that  the 
model  was  properly  implemented  within  MatLab™.  How¬ 
ever,  subsequent  application  of  the  model  for  A6082,  using 
a  modified  thermodynamic  relationship  provided  by  Myhr 
[17]  revealed  an  unexpected  anomalous  secondary  spike  in 


(  i 


Precipitate  Size  (Angstoms) 

Figure  5  Comparison  of  particle  size  distributions  calculated  by 
Myhr  [16]  (benchmark)  and  the  current  MatLab™ 
implementation  following  10  hours  age  at  180°C. 
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Figure  6  A6082  precipitate  size  distribution  exhibited  a 
transient  anomalous  peak  following  a  2  hour  aging  heat 
treatment  at  1 80°C. 

\ _ y 


the  predicted  particle  size  distribution.  This  peak,  shown 
in  Figure  6,  was  transient,  forming  and  disappearing  within 
a  relatively  brief  time  window.  The  presence  of  the  anom¬ 
alous  peak  required  additional  verification  that  involved 
determining  whether  it  reflected  a  numerical  instability,  a 
discontinuity  in  the  revised  thermodynamic  formulation, 
or  a  spurious  nucleation  event. 

Study  of  the  anomaly,  involving  changing  the  integration 
time  and  spatial  increments  along  with  use  of  the  earlier 
thermodynamic  relationship,  failed  to  eliminate  the  anom¬ 
alous  peak.  However,  careful  examination  of  mass  flows 
among  cells  in  the  vicinity  of  the  peak  and  inspection  of 
the  local  cell  mass  balance,  indicated  that  the  anomalous 
peak  was  a  natural  consequence  of  the  nucleation  formula¬ 
tion.  Specifically,  the  Myhr  model  assumes  that  all  nuclei 
forming  during  a  time  step  have  a  fixed  radius,  which  leads 
to  an  exaggerated  increase  in  number  density  for  that  pre¬ 
cipitate  size.  This  conclusion  was  confirmed  by  “turning 
or  nucleation  during  the  critical  time  window,  which 
eliminated  the  secondary  peak.  Despite  this  finding,  the 
team  decided  against  attempting  to  fix  the  nucleation  for¬ 
mulation  (e.g.,  by  dispersing  the  nuclei  radius)  because  no 
data  was  available  to  guide  or  validate  such  an  adjustment; 
and  furthermore,  the  spurious  peak  had  no  significant  ef¬ 
fect  on  yield  strength  predictions,  as  verified  via  analysis 
using  the  ancillary  strength  model. 

Validation  of  the  A6082  Precipitation  Model 

The  Myhr  research  team  [16,17]  had  rigorously  validated 
the  foundational  model  for  a  range  of  aluminum- 
magnesium-silicon  alloys  and  select  thermal  cycles.  This 
work  included  activities  such  as  electron  microscopy  to 
measure  precipitate  sizes  and  number  densities,  and 
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hardness  testing  to  infer  yield  strength.  Validation  and  un¬ 
certainty  assessment  of  the  A6082  MatLab  implementa¬ 
tion  was  significantly  more  modest  and  included  both 
experimental  and  modeling  activities.  In  overview,  model 
validation  consisted  of  comparing  simulation  of  precipita¬ 
tion  and  subsequent  hardening  against  experimentally 
determined  aging  curves  that  describe  A6082  hardness  as 
a  function  of  aging  time.  These  efforts,  listed  below, 
included  activities  described  in  sections  M2  and  M4  of  the 
V&V  checklist. 

•  Measurement  of  temperature  variation  during  heat 
treatment  and  hardness  measurement  error; 

•  Heat  treatment  trials  involving  the  aging  of  coupons 
for  differing  lengths  of  time; 

•  Replicated  hardness  measurements  for  each  coupon 
coupled  with  subsequent  calculation  of  hardness 
uncertainty; 

•  Simulation  of  precipitate  size  distributions  and 
associated  hardness  values  for  A6082  coupons  heat 
treated  per  the  experimental  plan;  and 

•  Synthesis  of  a  lower  bound  of  simulation  error  via 
the  Monte  Carlo  method  using  the  results  of  a 
sensitivity  analysis  and  uncertainty  estimates  for 
model  inputs  and  parameters.  The  results  were 
judged  to  represent  a  lower  bound  because  not  all 
sources  of  uncertainty  were  included  in  the  analysis. 

The  results  of  the  validation  exercise  and  embedded 
uncertainty  assessment  are  shown  in  Figure  7.  The  graph 
shows  that  the  A6082  precipitation  modeling  results  (in 
combination  with  the  hardness  model)  agree  well  with 
the  experimental  result  and  both  exhibit  comparable 
levels  of  uncertainty  for  the  alloy  and  heat  treatment 
used  in  this  study. 

TML  assessment  and  "risk  vs.  Consequences"  example 

As  with  the  ICME  V&V  example  described  in  the  previ¬ 
ous  section,  example  applications  of  the  Tool  Maturity 
Level  assessment  guide,  and  associated  risk  vs.  conse¬ 
quences  assessments,  are  needed  to  help  practitioners 
accelerate  and  strengthen  the  application  of  V&V  within 
ICME  development  programs.  Again,  the  best  and  most 
relevant  examples  will  ultimately  be  drawn  from  success¬ 
ful  ICME  projects  involving  an  integrated  system  of 
models  targeted  for  specific  applications.  A  hypothetical 
example  based  on  the  results  of  a  recent  Air  Force  pro¬ 
gram  to  develop  standardized  residual  stress  measure¬ 
ment  and  modeling  techniques  is  provided  below. 

Consider  a  comprehensive  ICME  effort  aimed  at  pre¬ 
dicting  bulk  residual  stresses  in  an  aerospace  component, 
which  will  include  process-induced  residual  stresses  from 
forging  and  heat  treatment,  and  the  redistribution  of 
these  stresses  after  final  machining.  The  goal  of  this 
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Figure  7  Comparison  of  experimental  and  modeling  validation  results  for  the  MatLab  Implementation  of  the  Myhr  precipitation  model. 

V  J 


ICME  program  is  to  reduce  the  machining  cost  by  pre¬ 
dicting  and  managing  any  distortion  within  the  desired 
final  part  envelop  and  to  integrate  the  analytically  pre¬ 
dicted  bulk  residual  stress  effects  into  final  component 
service-life  predictions.  Model  development  includes 
new  modules  as  well  as  the  extension  of  existing  tools  by 
incorporating  new  data  related  to  the  specific  material 
and  application.  V&V  plans  have  been  developed  in  ac¬ 
cordance  with  the  Checklists  with  the  intent  to  achieve 
TML-3  at  completion. 

TML-3  [4]  represents  a  mature  analytical  tool  or 
model  —  one  that  could  significantly  reduce  or  eliminate 
iterations  in  a  material  or  process  development  program, 
and  potentially  reduce  experimental  testing  or  other  ana¬ 
lytical  requirements.  At  TML-3,  the  model  can  be  used 
directly  for  assessments  or  evaluations  of  derivative 
materials  or  processes,  or  deviations  from  known  prac¬ 
tices.  While  the  accuracy  or  fidelity  of  the  model  may 
not  be  fully  validated,  the  range  of  application  should  be 
well  defined  and  documented.  In  addition,  supporting 
data  should  represent  the  entire  range  of  application,  a 
Users  Guide  should  be  fully  developed  and  updated 
(from  TML-2  level),  version  control  for  software  imple¬ 
mented,  and  significant  sub-model  validation  completed. 
Finally,  at  TML-3,  a  Risk  vs.  Consequences  assessment  is 
highly  recommended  prior  to  any  significant  application. 

In  this  hypothetical  example,  it  is  uncertain  whether 
TML-3  will  be  totally  achieved,  and  what  impact  that 
might  have  on  realizing  all  the  program  and  project 
goals.  Further,  it  is  uncertain  whether  TRL-3  is  an  ad¬ 
equate  maturity  level  to  achieve  the  most  ambitious 
project  goal:  use  of  the  predicted  residual  stress  fields  in 
the  component  life  predictions  for  fatigue  and  damage 
tolerance.  Consequently,  the  developers  and  project 


team  generated  a  high-level  flow  diagram  from  their 
System  and  Model  level  checklist  information,  to  help 
assess  the  critical  elements  for  V&V,  as  illustrated  in 
Figure  8. 

The  project  and  development  team  identified  five  spe¬ 
cific  items  that  appear  to  be  significant  risk  areas  that 
warrant  additional  consideration  in  the  V&V  plan  for 
this  application.  These  are  indicated  with  numbered 
symbols  in  Figure  8.  The  team  then  constructed  a  work¬ 
ing  table  to  list  the  specific  risks,  estimated  the  likelihood 
of  occurrence  and  the  severity  or  potential  impact  to  the 
project. 

The  developers  and  project  team  determined  risk 
mitigation  actions,  where  appropriate,  and  estimated 
the  effect  on  reducing  program  risk  and  potential 
consequences: 

1.  Thermal  process  modeling  must  accurately  predict 
temperatures  and  temperature  transients  during 
processing  in  order  to  predict  resulting  residual  stress 
fields.  The  project  team  performed  a  parametric 
analysis  where  such  predictions  were  evaluated  for 
typical  aerospace  rotating  components.  This  analysis 
found  that  determination  of  accurate  heat  transfer 
coefficients,  especially  for  use  during  high  transient 
thermal  processes,  was  critical.  Consequently, 
additional  measurements  of  part  temperature  during 
processing  and  use  of  complementary  methods  to 
estimate  heat  transfer  coefficients,  were  deemed 
necessary. 

2.  Measured  residual  stresses  must  reasonably  validate 
predicted  values  for  the  intended  purpose.  Virtual 
experiments  were  performed  with  combined  error 
sources  using  nominal  magnitudes  in  order  to 
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Figure  8  Simple  flow  map  for  residual  stress  hypothetical  example. 

estimate  the  current  stress  prediction  error  for  the 
residual  stress  measurement  technique.  In 
performing  the  virtual  experiments,  emphasis  was 
placed  on  simulating  "worst-case"  scenario  error 
sources  in  order  to  understand  the  lower  bounds  of 
accuracy  associated  with  the  method.  Based  on  the 
results  of  the  virtual  experiments  the  team  planned 
additional  stress-relaxation  mechanical  tests  to 
ensure  adequate  constitutive  data  to  support  the 
modeling  efforts.  The  team  also  decided  to  use 
multiple  methods  for  residual  stress  measurement 
validation. 

3.  Variation  in  final  microstructures  must  be  within 
desired  control  limits  for  the  intended  application  — 
to  ensure  that  material  properties  and  behavior  meet 
design  intent.  The  project  team  recognized  this  as  a 
potential  risk,  but  has  experience  with 
microstructure  evolution  modeling  for  this  material 
and  determined  that  the  risk  can  be  mitigated  or 
eliminated  with  early  production  monitoring  and 
controls. 

4.  Distortion  during  finish  machining  may  result  from 
redistribution  of  process-induced  residual  stresses . 

This  may  result  from  placement  of  the  final  part 
within  the  forging  envelope,  as  was  found  in  a  recent 
case  study  for  an  aerospace  aluminum  forging  [19]. 
For  the  rotating  parts  considered  in  this  project,  the 
team  regarded  this  risk  as  low  but  required 
simulation  of  final  part  placement  in  the  forging 
envelop,  and  assessment  of  machining  sequence 
effects  on  distortion. 

5.  Fatigue  and  damage  tolerance  life  predictions  require 
high  fidelity  stress  values  for  required  accuracy .  The 
project  team  was  concerned  that  the  accuracy  of 
residual  stress  predictions  may  be  suitable  for  use  in 


defining  forging  shapes,  reducing  input  weights,  and 
predicting  or  controlling  part  distortion  during 
machining,  but  not  sufficiently  accurate  to  use 
directly  in  life  predictions.  Further,  potential  errors 
in  residual  stress  measurement  accuracy  may 
preclude  or  limit  validation.  The  project  team 
regarded  this  as  a  risk  that  was  likely  to  occur  based 
on  similar  studies  [19],  and  that  would  have 
significant  impact  if  it  did  occur.  The  team  adjusted 
this  specific  project  goal  and  decided  to  use  the 
residual  stresses  for  static  assessments  such  as 
potential  impact  on  rotor  burst  limits,  rather  than 
directly  in  Low  Cycle  Fatigue  (LCF)  and  fracture 
mechanics  life  predictions.  The  team  agreed  to 
reassess  at  a  future  date,  pending  improved  elements 
of  the  ICME  modeling  efforts,  measurement 
techniques,  or  demonstrated  ability  to  achieve  TML- 
4  for  this  process. 

The  resulting  list  of  risks,  estimated  likelihood  of  oc¬ 
currence,  estimated  consequences,  and  a  simple  “red- 
yellow-green”  assessment  of  importance  are  summarized 
in  Figure  9. 

Expected  effects  of  the  planned  mitigation  items  are 
also  shown  in  Figure  9,  with  changes  indicated  by  the 
small  arrows.  The  five  risk  items  are  plotted  on  the  5X5 
Risk  vs.  Consequences  Matrix  in  Figure  10,  at  the  assessed 
levels  for  likelihood  and  consequence.  This  figure  illus¬ 
trates  the  effect  of  the  planned  mitigation  actions,  and 
improvements  to  the  V&V  plan  for  this  ICME  project. 
The  assessment  indicated  that  Items  1  and  3  could  be 
improved  from  “yellow”  to  “green”  with  mitigation 
actions,  and  that  Items  2  and  4  posed  some  risk  but  did 
not  require  any  risk  mitigation  actions.  Item  5  obviously 
posed  a  high  risk  which  could  be  reduced  with  mitigation 
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Item 

Risk 

Likelihood 

Consequence 

Rating 

i 

Risk-  Thermal  process  model  does  not  predict 

temperature  transients  accurately. 

3  (>  30%, 
Moderate) 

3  (Moderate) 

Y 

Impact:  Degrades  all  downstream  analyses. 

O 

, . .  Add  measurements  to  improve  heat  transfer 

predictions. 

2 

3 

G 

2 

Risk-  Stress  relief  predictions  do  not  match 

measured  forging  data  after  heat  treatment 

2  (>5%,  Low) 

3  (Moderate) 

G 

.  x  Some  effect  on  residual  stress  magnitude 

Impact:  ..  y 

predictions. 

M  itigation :  None  required:  monitor. 

3 

Risk-  Range  of  micros tructure  and  grain  size 

predictions  exceeds  desired  control  limits 

3  (>  30%, 
Moderate) 

2  (Some  impact) 

Y 

Impact:  May  affect  life  prediction  with  residual  stresses 

V 

M  itigation :  Extend  control  limits;  monitor  production. 

2  (>5%,  Low) 

2  (Some  impact) 

1  <3 

4 

Risk-  Machining  distortion  requires  cold-straightening 

and  stress  relief  to  meet  final  envelop 

2  (>5%,  Low) 

3  (Moderate) 

G 

Impact:  Extra  process  &  modeling  step,  but  predictable 

M  itigation :  None  required:  monitor. 

Risk-  Uncertainty  quantification  indicates  RS  values 

have  insufficent  fidelity  for  life  prediction 
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Figure  9  Example  of  risk  identification,  impact  and  mitigation  estimates. 
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Figure  10  Example  use  of  risk  matrix  and  effect  of  mitigation  actions. 
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actions  but  would  still  warrant  attention  of  the  project 
and  development  team. 

This  is  just  a  hypothetical  example,  of  course,  but  it 
shows  how  a  project  risk  management  tool  can  be  used 
in  conjunction  with  the  ICME  System  Level  and  Model 
V&V  Checklists,  and  the  TML  Assessment  guide  to  as¬ 
sist  ICME  V&V  planning,  especially  for  developmental 
ICME  models  or  methods,  or  when  development  pro¬ 
gram  decisions  will  be  ICME  based. 

Conclusions 

ICME  has  the  potential  to  greatly  benefit  the  materials 
science  and  engineering  communities,  and  to  greatly  en¬ 
hance  integration  with  other  engineering  disciplines.  The 
potential  benefits  of  ICME  to  reduce  time,  cost,  or  risk, 
and  to  enhance  future  “design”  of  materials  and  pro¬ 
cesses,  are  enormous.  Verification  and  validation  of 
models  and  methods  poses  a  significant  challenge  to 
broad  development,  implementation,  and  acceptance  of 
ICME.  This  is  especially  relevant  where  significant  deci¬ 
sions  will  be  ICME-based.  That  is,  where  decisions  may 
affect  technology  or  product  development,  legacy  system 
sustainment  actions,  supply  base  decisions,  or  quality 
assessments. 

The  approach  for  ICME  verification  and  validation 
described  in  this  paper  was  developed  based  on  a  philoso¬ 
phy  aligned  with  current  well-established  technology  and 
product  development  processes.  The  guidelines  are  con¬ 
sistent  with,  and  frequently  reference,  the  well-established 
ASME  V&V  Guide  10-2006,  developed  by  the  computa¬ 
tional  solid  mechanics  community  after  many  years  of  ef¬ 
fort.  Finally,  the  tools  that  were  developed  to  aid  ICME 
developers  include  an  ICME  V&V  System-Level  Check¬ 
list,  a  Model  V&V  Checklist,  a  TML  Assessment  Guide, 
and  a  risk  matrix  tool  for  project  risk  assessment  and 
mitigation.  Much  effort  has  gone  into  making  these  tools 
broadly  applicable,  useful,  simple,  and  flexible  —  and 
then  integrating  them  to  ensure  compatibility.  Instruc¬ 
tions  for  their  use,  and  initial  examples  of  their  applica¬ 
tion  were  developed. 

The  authors  recognize  that  this  is  and  should  be  an 
evolving  effort.  The  tools  and  approach  outlined  here 
will  be  continuously  improved  and  updated  with  increas¬ 
ing  application  experience. 
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